{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here I will extract and engineer the features that will be the input for a subsequent model.\n",
    "\n",
    "The main data are of the follwoing format:\n",
    "\n",
    "* The data report weather measurements for ~1000 unique locations in ~150 counties across 5 states. \n",
    "* Most locations have one entry per day reporting the current weather conditions.\n",
    "* At the end of the season, the harvest generates a certain yield. This yield is propagated to *all* entries in the data set, even though it is only a final value.\n",
    "* The reported yield number refers to the yield in the county and is not specific to a location. It is unclear if it is an average or a sum for the county.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "My goal here is to create a profile across the season for each location from the avaiable plus additional data. This will leave me with one set of feature values for each location, which is connected to a final yield. And that will be the input to my model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pickle\n",
    "import time\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.basemap import Basemap\n",
    "\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def find_30day_extreme(in_df, in_column, in_agg='min'):\n",
    "    \"\"\"\n",
    "    Calculates a rolling window mean. Then determines the min/max/mean \n",
    "    value of all the window aggregates.\n",
    "    Window size: 30 days.\n",
    "    in_df: input dataframe\n",
    "    in_column: column to be aggregated\n",
    "    in_agg: aggregation function for the list of results ('min','max','mean')\n",
    "    \"\"\"\n",
    "    # Pandas 30 day rolling window. Make sure there are at least 10 samples in the window.\n",
    "    agg_df = in_df[[in_column,'Date']].rolling('30d', on='Date', min_periods=10).mean()\n",
    "    agg_list = agg_df[in_column]\n",
    "    if in_agg == 'min':\n",
    "        return np.min(agg_list)\n",
    "    if in_agg == 'max':\n",
    "        return np.max(agg_list)\n",
    "    if in_agg == 'mean':\n",
    "        return np.mean(agg_list)\n",
    " \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def calculate_features(tmp_df):\n",
    "    result = []\n",
    "    # Minimum average temperature in 30-day period\n",
    "    result.append(find_30day_extreme(tmp_df, 'temperatureAverage', in_agg='min'))\n",
    "    # Maximum average temperature in 30-day period\n",
    "    result.append(find_30day_extreme(tmp_df, 'temperatureAverage', in_agg='max'))\n",
    "    # Minimum NDVI in 30-day period\n",
    "    tmp1 = find_30day_extreme(tmp_df, 'temperatureAverage', in_agg='min')\n",
    "    # Maximum NDVI in 30-day period\n",
    "    tmp2 = find_30day_extreme(tmp_df, 'temperatureAverage', in_agg='max')\n",
    "    result.append(tmp2/tmp1)\n",
    "    # Mean temperature difference and variance\n",
    "    result.append(tmp_df['temperatureDiff'].mean())\n",
    "    result.append(tmp_df['temperatureDiff'].std())\n",
    "    # Mean wind speed\n",
    "    result.append(tmp_df['windSpeed'].mean())\n",
    "    # Total precipitation\n",
    "    result.append(tmp_df['precipTotal'].max())\n",
    "    # Total yield\n",
    "    result.append(tmp_df['Yield'].max())\n",
    "    #\n",
    "    return result\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These data have been slighlty pre-processed. See 01_data_exploration.ipynb for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cwd = os.getcwd()\n",
    "df_2013 = pd.read_pickle(os.path.join(cwd,'data','df_2013_clean.df'))\n",
    "df_2014 = pd.read_pickle(os.path.join(cwd,'data','df_2014_clean.df'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have also obtained information on the elevation and length_of_day for each location (see 03_elevation_and_length_of_day.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Load data - dictionaries\n",
    "with open(os.path.join('data','elevation.pickle'), 'rb') as handle:\n",
    "    elevation = pickle.load(handle)\n",
    "with open(os.path.join('data','length_of_day.pickle'), 'rb') as handle:\n",
    "    length_of_day = pickle.load(handle)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each location comes with:\n",
    "\n",
    "* longitude\n",
    "* latitude\n",
    "\n",
    "The features I am going to engineer for each location are:\n",
    "\n",
    "* the total amount of precipitation\n",
    "* the minimum average temperature in a consecutive 30-day period\n",
    "* the maximum average temperature in a consecutive 30-day period\n",
    "* ratio of maximum average NDVI in a consecutive 30-day period and its respective minimum\n",
    "* the mean temperature difference between daily min/max temperatures\n",
    "* the standard deviation of the above mean\n",
    "* the total average wind speed\n",
    "\n",
    "I will add to those features external values of:\n",
    "\n",
    "* the hours of daylight\n",
    "* the elevation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During the engineering I will keep both years (2013 and 2014) separate. Even though these data cover mostly the same locations (~>80% overlap), weather conditions and yield is likely to be different. Keeping them separate provides additional data points. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exclude locations with only very few measurements across the growing period"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since I am aggregating data across the growing period, I will for this current approach remove locations with only very few records. For these locations it is difficult to generate the features above in a straight forward manner. (See 01_data_exploration.ipynb for details on how these locations were identified)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1014\n",
      "955\n",
      "1035\n",
      "982\n"
     ]
    }
   ],
   "source": [
    "locs = df_2013['Location'].unique().tolist()\n",
    "print(len(locs))\n",
    "n_records_2013 = df_2013.groupby(by='Location').agg({'Location': {'Count' : lambda x: x.count()}})\n",
    "n_records_2013['Location']['Count'].value_counts().sort_index(ascending=False)\n",
    "klocs = n_records_2013[n_records_2013['Location']['Count'] >= 153].index.get_level_values('Location').values\n",
    "locs_2013 = df_2013['Location'][df_2013['Location'].isin(klocs)].unique().tolist()\n",
    "print(len(locs_2013))\n",
    "#\n",
    "locs = df_2014['Location'].unique().tolist()\n",
    "print(len(locs))\n",
    "\n",
    "n_records_2014 = df_2014.groupby(by='Location').agg({'Location': {'Count' : lambda x: x.count()}})\n",
    "n_records_2014['Location']['Count'].value_counts().sort_index(ascending=False)\n",
    "klocs = n_records_2014[n_records_2014['Location']['Count'] >= 153].index.get_level_values('Location').values\n",
    "locs_2014 = df_2014['Location'][df_2014['Location'].isin(klocs)].unique().tolist()\n",
    "print(len(locs_2014))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2013"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CountyName</th>\n",
       "      <th>State</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Date</th>\n",
       "      <th>cloudCover</th>\n",
       "      <th>dewPoint</th>\n",
       "      <th>humidity</th>\n",
       "      <th>precipIntensity</th>\n",
       "      <th>precipProbability</th>\n",
       "      <th>...</th>\n",
       "      <th>windBearing</th>\n",
       "      <th>windSpeed</th>\n",
       "      <th>NDVI</th>\n",
       "      <th>DayInSeason</th>\n",
       "      <th>Yield</th>\n",
       "      <th>Location</th>\n",
       "      <th>precipTotal</th>\n",
       "      <th>temperatureDiff</th>\n",
       "      <th>temperatureRatio</th>\n",
       "      <th>temperatureAverage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Adams</td>\n",
       "      <td>Washington</td>\n",
       "      <td>46.811686</td>\n",
       "      <td>-118.695237</td>\n",
       "      <td>2013-11-30</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.53</td>\n",
       "      <td>0.91</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>...</td>\n",
       "      <td>214</td>\n",
       "      <td>1.18</td>\n",
       "      <td>134.110657</td>\n",
       "      <td>0</td>\n",
       "      <td>35.7</td>\n",
       "      <td>(-118.6952372, 46.8116858)</td>\n",
       "      <td>0.000</td>\n",
       "      <td>8.22</td>\n",
       "      <td>1.299127</td>\n",
       "      <td>31.590</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Adams</td>\n",
       "      <td>Washington</td>\n",
       "      <td>46.929839</td>\n",
       "      <td>-118.352109</td>\n",
       "      <td>2013-11-30</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.77</td>\n",
       "      <td>0.93</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.05</td>\n",
       "      <td>...</td>\n",
       "      <td>166</td>\n",
       "      <td>1.01</td>\n",
       "      <td>131.506592</td>\n",
       "      <td>0</td>\n",
       "      <td>35.7</td>\n",
       "      <td>(-118.3521093, 46.9298391)</td>\n",
       "      <td>0.000</td>\n",
       "      <td>8.18</td>\n",
       "      <td>1.303863</td>\n",
       "      <td>31.010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Adams</td>\n",
       "      <td>Washington</td>\n",
       "      <td>47.006888</td>\n",
       "      <td>-118.510160</td>\n",
       "      <td>2013-11-30</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.36</td>\n",
       "      <td>0.94</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.06</td>\n",
       "      <td>...</td>\n",
       "      <td>158</td>\n",
       "      <td>1.03</td>\n",
       "      <td>131.472946</td>\n",
       "      <td>0</td>\n",
       "      <td>35.7</td>\n",
       "      <td>(-118.5101603, 47.0068881)</td>\n",
       "      <td>0.020</td>\n",
       "      <td>6.43</td>\n",
       "      <td>1.238590</td>\n",
       "      <td>30.165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Adams</td>\n",
       "      <td>Washington</td>\n",
       "      <td>47.162342</td>\n",
       "      <td>-118.699677</td>\n",
       "      <td>2013-11-30</td>\n",
       "      <td>0.91</td>\n",
       "      <td>29.47</td>\n",
       "      <td>0.94</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.15</td>\n",
       "      <td>...</td>\n",
       "      <td>153</td>\n",
       "      <td>1.84</td>\n",
       "      <td>131.288300</td>\n",
       "      <td>0</td>\n",
       "      <td>35.7</td>\n",
       "      <td>(-118.6996774, 47.1623419)</td>\n",
       "      <td>0.036</td>\n",
       "      <td>6.02</td>\n",
       "      <td>1.221568</td>\n",
       "      <td>30.180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Adams</td>\n",
       "      <td>Washington</td>\n",
       "      <td>47.157512</td>\n",
       "      <td>-118.434056</td>\n",
       "      <td>2013-11-30</td>\n",
       "      <td>0.91</td>\n",
       "      <td>29.86</td>\n",
       "      <td>0.94</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>0.24</td>\n",
       "      <td>...</td>\n",
       "      <td>156</td>\n",
       "      <td>1.85</td>\n",
       "      <td>131.288300</td>\n",
       "      <td>0</td>\n",
       "      <td>35.7</td>\n",
       "      <td>(-118.4340559, 47.157512)</td>\n",
       "      <td>0.000</td>\n",
       "      <td>6.78</td>\n",
       "      <td>1.250462</td>\n",
       "      <td>30.460</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  CountyName       State   Latitude   Longitude       Date  cloudCover  \\\n",
       "0      Adams  Washington  46.811686 -118.695237 2013-11-30        0.00   \n",
       "1      Adams  Washington  46.929839 -118.352109 2013-11-30        0.00   \n",
       "2      Adams  Washington  47.006888 -118.510160 2013-11-30        0.00   \n",
       "3      Adams  Washington  47.162342 -118.699677 2013-11-30        0.91   \n",
       "4      Adams  Washington  47.157512 -118.434056 2013-11-30        0.91   \n",
       "\n",
       "   dewPoint  humidity  precipIntensity  precipProbability         ...          \\\n",
       "0     29.53      0.91           0.0000               0.00         ...           \n",
       "1     29.77      0.93           0.0001               0.05         ...           \n",
       "2     29.36      0.94           0.0001               0.06         ...           \n",
       "3     29.47      0.94           0.0002               0.15         ...           \n",
       "4     29.86      0.94           0.0003               0.24         ...           \n",
       "\n",
       "   windBearing  windSpeed        NDVI  DayInSeason  Yield  \\\n",
       "0          214       1.18  134.110657            0   35.7   \n",
       "1          166       1.01  131.506592            0   35.7   \n",
       "2          158       1.03  131.472946            0   35.7   \n",
       "3          153       1.84  131.288300            0   35.7   \n",
       "4          156       1.85  131.288300            0   35.7   \n",
       "\n",
       "                     Location  precipTotal  temperatureDiff  temperatureRatio  \\\n",
       "0  (-118.6952372, 46.8116858)        0.000             8.22          1.299127   \n",
       "1  (-118.3521093, 46.9298391)        0.000             8.18          1.303863   \n",
       "2  (-118.5101603, 47.0068881)        0.020             6.43          1.238590   \n",
       "3  (-118.6996774, 47.1623419)        0.036             6.02          1.221568   \n",
       "4   (-118.4340559, 47.157512)        0.000             6.78          1.250462   \n",
       "\n",
       "   temperatureAverage  \n",
       "0              31.590  \n",
       "1              31.010  \n",
       "2              30.165  \n",
       "3              30.180  \n",
       "4              30.460  \n",
       "\n",
       "[5 rows x 27 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_2013.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features = ['longitude',\n",
    "            'latitude',\n",
    "            'elevation',\n",
    "            'LOD',\n",
    "            'total_precipitation',\n",
    "            'minMAT30',\n",
    "            'maxMAT30',\n",
    "            'ratioMNDVI30',\n",
    "            'mean_wind_speed',\n",
    "            'mean_temperature_diff',\n",
    "            'std_temperature_diff',\n",
    "            'yield',\n",
    "           ]\n",
    "\n",
    "df_2013_new = pd.DataFrame(columns=features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Exec. time: 37.78 s\n"
     ]
    }
   ],
   "source": [
    "now = time.time()\n",
    "\n",
    "for idx,loc in enumerate(locs_2013):\n",
    "    tmp_df = df_2013[df_2013['Location'] == loc]\n",
    "    (min_mean_average_temparature_30days, \n",
    "     max_mean_average_temparature_30days,\n",
    "     ratio_mean_ndvi_30days,\n",
    "     mean_temperature_diff,\n",
    "     std_temperature_diff,\n",
    "     mean_wind_speed,\n",
    "     total_precipitation,\n",
    "     total_yield) = calculate_features(tmp_df)\n",
    "    try:\n",
    "        new_elevation = elevation[loc]\n",
    "    except:\n",
    "        print('Elevation: no match for location found', loc)\n",
    "    try:\n",
    "        new_length_of_day = length_of_day[loc]\n",
    "    except:\n",
    "        print('Length-of-day: no match for location found', loc)\n",
    "    longitude = loc[0]\n",
    "    latitude = loc[1]\n",
    "    observations = {'longitude': longitude, \n",
    "        'latitude': latitude,\n",
    "        'elevation': new_elevation,\n",
    "        'LOD': new_length_of_day,\n",
    "        'total_precipitation': total_precipitation,\n",
    "        'minMAT30': min_mean_average_temparature_30days,\n",
    "        'maxMAT30': max_mean_average_temparature_30days,\n",
    "        'ratioMNDVI30': ratio_mean_ndvi_30days,\n",
    "        'mean_wind_speed': mean_wind_speed,\n",
    "        'mean_temperature_diff': mean_temperature_diff,\n",
    "        'std_temperature_diff': std_temperature_diff,\n",
    "        'yield': total_yield,\n",
    "       }\n",
    "    df_2013_new.loc[idx] = pd.Series(observations)\n",
    "        \n",
    "print('Exec. time: {:5.2f} s'.format(time.time()-now))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "955"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(locs_2013)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(955, 12)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_2013_new.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>elevation</th>\n",
       "      <th>LOD</th>\n",
       "      <th>total_precipitation</th>\n",
       "      <th>minMAT30</th>\n",
       "      <th>maxMAT30</th>\n",
       "      <th>ratioMNDVI30</th>\n",
       "      <th>mean_wind_speed</th>\n",
       "      <th>mean_temperature_diff</th>\n",
       "      <th>std_temperature_diff</th>\n",
       "      <th>yield</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-118.695237</td>\n",
       "      <td>46.811686</td>\n",
       "      <td>427.179199</td>\n",
       "      <td>11.171944</td>\n",
       "      <td>9.982</td>\n",
       "      <td>22.090000</td>\n",
       "      <td>62.033000</td>\n",
       "      <td>2.808194</td>\n",
       "      <td>5.298548</td>\n",
       "      <td>17.294409</td>\n",
       "      <td>9.061316</td>\n",
       "      <td>35.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-118.352109</td>\n",
       "      <td>46.929839</td>\n",
       "      <td>524.561829</td>\n",
       "      <td>11.168056</td>\n",
       "      <td>12.312</td>\n",
       "      <td>21.140909</td>\n",
       "      <td>60.183833</td>\n",
       "      <td>2.846795</td>\n",
       "      <td>5.594247</td>\n",
       "      <td>16.485699</td>\n",
       "      <td>8.485870</td>\n",
       "      <td>35.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-118.510160</td>\n",
       "      <td>47.006888</td>\n",
       "      <td>499.476074</td>\n",
       "      <td>11.165833</td>\n",
       "      <td>13.750</td>\n",
       "      <td>20.644545</td>\n",
       "      <td>59.363000</td>\n",
       "      <td>2.875481</td>\n",
       "      <td>6.132742</td>\n",
       "      <td>16.913817</td>\n",
       "      <td>8.444363</td>\n",
       "      <td>35.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-118.699677</td>\n",
       "      <td>47.162342</td>\n",
       "      <td>521.150330</td>\n",
       "      <td>11.160833</td>\n",
       "      <td>12.934</td>\n",
       "      <td>20.151818</td>\n",
       "      <td>59.682000</td>\n",
       "      <td>2.961619</td>\n",
       "      <td>6.120430</td>\n",
       "      <td>16.800269</td>\n",
       "      <td>8.512496</td>\n",
       "      <td>35.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-118.434056</td>\n",
       "      <td>47.157512</td>\n",
       "      <td>571.114075</td>\n",
       "      <td>11.161111</td>\n",
       "      <td>16.644</td>\n",
       "      <td>20.138182</td>\n",
       "      <td>58.143167</td>\n",
       "      <td>2.887210</td>\n",
       "      <td>6.055968</td>\n",
       "      <td>16.077849</td>\n",
       "      <td>8.201818</td>\n",
       "      <td>35.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-118.958859</td>\n",
       "      <td>47.150327</td>\n",
       "      <td>451.084656</td>\n",
       "      <td>11.161111</td>\n",
       "      <td>8.905</td>\n",
       "      <td>21.740833</td>\n",
       "      <td>61.464167</td>\n",
       "      <td>2.827130</td>\n",
       "      <td>5.320806</td>\n",
       "      <td>17.446237</td>\n",
       "      <td>8.471867</td>\n",
       "      <td>35.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-98.330851</td>\n",
       "      <td>36.995835</td>\n",
       "      <td>391.887451</td>\n",
       "      <td>11.439722</td>\n",
       "      <td>47.256</td>\n",
       "      <td>28.731667</td>\n",
       "      <td>71.526500</td>\n",
       "      <td>2.489466</td>\n",
       "      <td>9.218011</td>\n",
       "      <td>24.476720</td>\n",
       "      <td>9.612832</td>\n",
       "      <td>14.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-98.404629</td>\n",
       "      <td>36.988813</td>\n",
       "      <td>401.383514</td>\n",
       "      <td>11.439722</td>\n",
       "      <td>51.077</td>\n",
       "      <td>28.703833</td>\n",
       "      <td>71.522333</td>\n",
       "      <td>2.491735</td>\n",
       "      <td>9.210806</td>\n",
       "      <td>24.486882</td>\n",
       "      <td>9.634120</td>\n",
       "      <td>14.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-98.319893</td>\n",
       "      <td>36.702452</td>\n",
       "      <td>359.771393</td>\n",
       "      <td>11.446389</td>\n",
       "      <td>55.124</td>\n",
       "      <td>29.362167</td>\n",
       "      <td>72.147500</td>\n",
       "      <td>2.457159</td>\n",
       "      <td>9.504355</td>\n",
       "      <td>24.687957</td>\n",
       "      <td>9.656204</td>\n",
       "      <td>14.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>-98.539816</td>\n",
       "      <td>36.628781</td>\n",
       "      <td>413.238281</td>\n",
       "      <td>11.448056</td>\n",
       "      <td>67.264</td>\n",
       "      <td>29.471167</td>\n",
       "      <td>72.077000</td>\n",
       "      <td>2.445679</td>\n",
       "      <td>9.398011</td>\n",
       "      <td>24.507634</td>\n",
       "      <td>9.583744</td>\n",
       "      <td>14.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    longitude   latitude   elevation        LOD  total_precipitation  \\\n",
       "0 -118.695237  46.811686  427.179199  11.171944                9.982   \n",
       "1 -118.352109  46.929839  524.561829  11.168056               12.312   \n",
       "2 -118.510160  47.006888  499.476074  11.165833               13.750   \n",
       "3 -118.699677  47.162342  521.150330  11.160833               12.934   \n",
       "4 -118.434056  47.157512  571.114075  11.161111               16.644   \n",
       "5 -118.958859  47.150327  451.084656  11.161111                8.905   \n",
       "6  -98.330851  36.995835  391.887451  11.439722               47.256   \n",
       "7  -98.404629  36.988813  401.383514  11.439722               51.077   \n",
       "8  -98.319893  36.702452  359.771393  11.446389               55.124   \n",
       "9  -98.539816  36.628781  413.238281  11.448056               67.264   \n",
       "\n",
       "    minMAT30   maxMAT30  ratioMNDVI30  mean_wind_speed  mean_temperature_diff  \\\n",
       "0  22.090000  62.033000      2.808194         5.298548              17.294409   \n",
       "1  21.140909  60.183833      2.846795         5.594247              16.485699   \n",
       "2  20.644545  59.363000      2.875481         6.132742              16.913817   \n",
       "3  20.151818  59.682000      2.961619         6.120430              16.800269   \n",
       "4  20.138182  58.143167      2.887210         6.055968              16.077849   \n",
       "5  21.740833  61.464167      2.827130         5.320806              17.446237   \n",
       "6  28.731667  71.526500      2.489466         9.218011              24.476720   \n",
       "7  28.703833  71.522333      2.491735         9.210806              24.486882   \n",
       "8  29.362167  72.147500      2.457159         9.504355              24.687957   \n",
       "9  29.471167  72.077000      2.445679         9.398011              24.507634   \n",
       "\n",
       "   std_temperature_diff  yield  \n",
       "0              9.061316   35.7  \n",
       "1              8.485870   35.7  \n",
       "2              8.444363   35.7  \n",
       "3              8.512496   35.7  \n",
       "4              8.201818   35.7  \n",
       "5              8.471867   35.7  \n",
       "6              9.612832   14.4  \n",
       "7              9.634120   14.4  \n",
       "8              9.656204   14.4  \n",
       "9              9.583744   14.4  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_2013_new.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2014"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CountyName</th>\n",
       "      <th>State</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Date</th>\n",
       "      <th>cloudCover</th>\n",
       "      <th>dewPoint</th>\n",
       "      <th>humidity</th>\n",
       "      <th>precipIntensity</th>\n",
       "      <th>precipProbability</th>\n",
       "      <th>...</th>\n",
       "      <th>windBearing</th>\n",
       "      <th>windSpeed</th>\n",
       "      <th>NDVI</th>\n",
       "      <th>DayInSeason</th>\n",
       "      <th>Yield</th>\n",
       "      <th>Location</th>\n",
       "      <th>precipTotal</th>\n",
       "      <th>temperatureDiff</th>\n",
       "      <th>temperatureRatio</th>\n",
       "      <th>temperatureAverage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Adams</td>\n",
       "      <td>Washington</td>\n",
       "      <td>46.929839</td>\n",
       "      <td>-118.352109</td>\n",
       "      <td>2014-11-30</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.77</td>\n",
       "      <td>0.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>3.80</td>\n",
       "      <td>136.179718</td>\n",
       "      <td>0</td>\n",
       "      <td>35.6</td>\n",
       "      <td>(-118.3521093, 46.9298391)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.97</td>\n",
       "      <td>3.438218</td>\n",
       "      <td>15.445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Adams</td>\n",
       "      <td>Washington</td>\n",
       "      <td>47.150327</td>\n",
       "      <td>-118.958859</td>\n",
       "      <td>2014-11-30</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.66</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>352</td>\n",
       "      <td>6.03</td>\n",
       "      <td>135.697540</td>\n",
       "      <td>0</td>\n",
       "      <td>35.6</td>\n",
       "      <td>(-118.9588592, 47.1503267)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.17</td>\n",
       "      <td>2.971297</td>\n",
       "      <td>17.295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Adams</td>\n",
       "      <td>Washington</td>\n",
       "      <td>46.811686</td>\n",
       "      <td>-118.695237</td>\n",
       "      <td>2014-11-30</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.55</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25</td>\n",
       "      <td>3.59</td>\n",
       "      <td>135.676956</td>\n",
       "      <td>0</td>\n",
       "      <td>35.6</td>\n",
       "      <td>(-118.6952372, 46.8116858)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.41</td>\n",
       "      <td>2.986683</td>\n",
       "      <td>16.465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Adams</td>\n",
       "      <td>Washington</td>\n",
       "      <td>47.162342</td>\n",
       "      <td>-118.699677</td>\n",
       "      <td>2014-11-30</td>\n",
       "      <td>0.03</td>\n",
       "      <td>7.32</td>\n",
       "      <td>0.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>5.18</td>\n",
       "      <td>135.005798</td>\n",
       "      <td>0</td>\n",
       "      <td>35.6</td>\n",
       "      <td>(-118.6996774, 47.1623419)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.38</td>\n",
       "      <td>3.145679</td>\n",
       "      <td>16.790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Adams</td>\n",
       "      <td>Washington</td>\n",
       "      <td>47.157512</td>\n",
       "      <td>-118.434056</td>\n",
       "      <td>2014-11-30</td>\n",
       "      <td>0.04</td>\n",
       "      <td>7.62</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>4.69</td>\n",
       "      <td>134.803864</td>\n",
       "      <td>0</td>\n",
       "      <td>35.6</td>\n",
       "      <td>(-118.4340559, 47.157512)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.51</td>\n",
       "      <td>2.984375</td>\n",
       "      <td>16.575</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  CountyName       State   Latitude   Longitude       Date  cloudCover  \\\n",
       "0      Adams  Washington  46.929839 -118.352109 2014-11-30        0.00   \n",
       "1      Adams  Washington  47.150327 -118.958859 2014-11-30        0.00   \n",
       "2      Adams  Washington  46.811686 -118.695237 2014-11-30        0.00   \n",
       "3      Adams  Washington  47.162342 -118.699677 2014-11-30        0.03   \n",
       "4      Adams  Washington  47.157512 -118.434056 2014-11-30        0.04   \n",
       "\n",
       "   dewPoint  humidity  precipIntensity  precipProbability         ...          \\\n",
       "0      6.77      0.69              0.0                0.0         ...           \n",
       "1      6.66      0.65              0.0                0.0         ...           \n",
       "2      6.55      0.67              0.0                0.0         ...           \n",
       "3      7.32      0.69              0.0                0.0         ...           \n",
       "4      7.62      0.70              0.0                0.0         ...           \n",
       "\n",
       "   windBearing  windSpeed        NDVI  DayInSeason  Yield  \\\n",
       "0            9       3.80  136.179718            0   35.6   \n",
       "1          352       6.03  135.697540            0   35.6   \n",
       "2           25       3.59  135.676956            0   35.6   \n",
       "3            1       5.18  135.005798            0   35.6   \n",
       "4            5       4.69  134.803864            0   35.6   \n",
       "\n",
       "                     Location  precipTotal  temperatureDiff  temperatureRatio  \\\n",
       "0  (-118.3521093, 46.9298391)          0.0            16.97          3.438218   \n",
       "1  (-118.9588592, 47.1503267)          0.0            17.17          2.971297   \n",
       "2  (-118.6952372, 46.8116858)          0.0            16.41          2.986683   \n",
       "3  (-118.6996774, 47.1623419)          0.0            17.38          3.145679   \n",
       "4   (-118.4340559, 47.157512)          0.0            16.51          2.984375   \n",
       "\n",
       "   temperatureAverage  \n",
       "0              15.445  \n",
       "1              17.295  \n",
       "2              16.465  \n",
       "3              16.790  \n",
       "4              16.575  \n",
       "\n",
       "[5 rows x 27 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_2014.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features = ['longitude',\n",
    "            'latitude',\n",
    "            'elevation',\n",
    "            'LOD',\n",
    "            'total_precipitation',\n",
    "            'minMAT30',\n",
    "            'maxMAT30',\n",
    "            'ratioMNDVI30',\n",
    "            'mean_wind_speed',\n",
    "            'mean_temperature_diff',\n",
    "            'std_temperature_diff',\n",
    "            'yield',\n",
    "           ]\n",
    "\n",
    "df_2014_new = pd.DataFrame(columns=features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Exec. time: 39.80 s\n"
     ]
    }
   ],
   "source": [
    "now = time.time()\n",
    "\n",
    "for idx,loc in enumerate(locs_2014):\n",
    "    tmp_df = df_2014[df_2014['Location'] == loc]\n",
    "    (min_mean_average_temparature_30days, \n",
    "     max_mean_average_temparature_30days,\n",
    "     ratio_mean_ndvi_30days,\n",
    "     mean_temperature_diff,\n",
    "     std_temperature_diff,\n",
    "     mean_wind_speed,\n",
    "     total_precipitation,\n",
    "     total_yield) = calculate_features(tmp_df)\n",
    "    try:\n",
    "        new_elevation = elevation[loc]\n",
    "    except:\n",
    "        print('Elevation: no match for location found', loc)\n",
    "    try:\n",
    "        new_length_of_day = length_of_day[loc]\n",
    "    except:\n",
    "        print('Length-of-day: no match for location found', loc)\n",
    "    longitude = loc[0]\n",
    "    latitude = loc[1]\n",
    "    observations = {'longitude': longitude, \n",
    "        'latitude': latitude,\n",
    "        'elevation': new_elevation,\n",
    "        'LOD': new_length_of_day,\n",
    "        'total_precipitation': total_precipitation,\n",
    "        'minMAT30': min_mean_average_temparature_30days,\n",
    "        'maxMAT30': max_mean_average_temparature_30days,\n",
    "        'ratioMNDVI30': ratio_mean_ndvi_30days,\n",
    "        'mean_wind_speed': mean_wind_speed,\n",
    "        'mean_temperature_diff': mean_temperature_diff,\n",
    "        'std_temperature_diff': std_temperature_diff,\n",
    "        'yield': total_yield,\n",
    "       }\n",
    "    df_2014_new.loc[idx] = pd.Series(observations)\n",
    "        \n",
    "print('Exec. time: {:5.2f} s'.format(time.time()-now))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "982"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(locs_2014)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(982, 12)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_2014_new.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>elevation</th>\n",
       "      <th>LOD</th>\n",
       "      <th>total_precipitation</th>\n",
       "      <th>minMAT30</th>\n",
       "      <th>maxMAT30</th>\n",
       "      <th>ratioMNDVI30</th>\n",
       "      <th>mean_wind_speed</th>\n",
       "      <th>mean_temperature_diff</th>\n",
       "      <th>std_temperature_diff</th>\n",
       "      <th>yield</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-118.352109</td>\n",
       "      <td>46.929839</td>\n",
       "      <td>524.561829</td>\n",
       "      <td>11.168056</td>\n",
       "      <td>3.362</td>\n",
       "      <td>29.588000</td>\n",
       "      <td>62.185833</td>\n",
       "      <td>2.101725</td>\n",
       "      <td>4.202903</td>\n",
       "      <td>17.679355</td>\n",
       "      <td>9.148266</td>\n",
       "      <td>35.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-118.958859</td>\n",
       "      <td>47.150327</td>\n",
       "      <td>451.084656</td>\n",
       "      <td>11.161111</td>\n",
       "      <td>1.536</td>\n",
       "      <td>29.681000</td>\n",
       "      <td>63.494667</td>\n",
       "      <td>2.139236</td>\n",
       "      <td>4.044194</td>\n",
       "      <td>17.873925</td>\n",
       "      <td>9.295265</td>\n",
       "      <td>35.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-118.695237</td>\n",
       "      <td>46.811686</td>\n",
       "      <td>427.179199</td>\n",
       "      <td>11.171944</td>\n",
       "      <td>2.472</td>\n",
       "      <td>30.269500</td>\n",
       "      <td>63.789667</td>\n",
       "      <td>2.107391</td>\n",
       "      <td>3.991398</td>\n",
       "      <td>18.457634</td>\n",
       "      <td>9.581113</td>\n",
       "      <td>35.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-118.699677</td>\n",
       "      <td>47.162342</td>\n",
       "      <td>521.150330</td>\n",
       "      <td>11.160833</td>\n",
       "      <td>3.528</td>\n",
       "      <td>29.487500</td>\n",
       "      <td>62.002333</td>\n",
       "      <td>2.102665</td>\n",
       "      <td>4.630054</td>\n",
       "      <td>17.826935</td>\n",
       "      <td>9.166665</td>\n",
       "      <td>35.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-118.434056</td>\n",
       "      <td>47.157512</td>\n",
       "      <td>571.114075</td>\n",
       "      <td>11.161111</td>\n",
       "      <td>4.317</td>\n",
       "      <td>28.834833</td>\n",
       "      <td>60.857667</td>\n",
       "      <td>2.110561</td>\n",
       "      <td>4.717097</td>\n",
       "      <td>17.187527</td>\n",
       "      <td>8.905124</td>\n",
       "      <td>35.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-118.510160</td>\n",
       "      <td>47.006888</td>\n",
       "      <td>499.476074</td>\n",
       "      <td>11.165833</td>\n",
       "      <td>3.671</td>\n",
       "      <td>29.496000</td>\n",
       "      <td>61.706000</td>\n",
       "      <td>2.092012</td>\n",
       "      <td>4.679677</td>\n",
       "      <td>17.844516</td>\n",
       "      <td>9.182010</td>\n",
       "      <td>35.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-98.454318</td>\n",
       "      <td>36.508221</td>\n",
       "      <td>394.365906</td>\n",
       "      <td>11.451111</td>\n",
       "      <td>10.565</td>\n",
       "      <td>31.529167</td>\n",
       "      <td>65.112000</td>\n",
       "      <td>2.065135</td>\n",
       "      <td>8.180699</td>\n",
       "      <td>20.610538</td>\n",
       "      <td>9.833779</td>\n",
       "      <td>29.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-98.444745</td>\n",
       "      <td>36.650698</td>\n",
       "      <td>401.645874</td>\n",
       "      <td>11.447778</td>\n",
       "      <td>13.368</td>\n",
       "      <td>31.417333</td>\n",
       "      <td>65.034000</td>\n",
       "      <td>2.070004</td>\n",
       "      <td>8.170323</td>\n",
       "      <td>20.738495</td>\n",
       "      <td>9.912246</td>\n",
       "      <td>29.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-98.319893</td>\n",
       "      <td>36.702452</td>\n",
       "      <td>359.771393</td>\n",
       "      <td>11.446389</td>\n",
       "      <td>12.767</td>\n",
       "      <td>31.339000</td>\n",
       "      <td>65.082167</td>\n",
       "      <td>2.076715</td>\n",
       "      <td>8.287366</td>\n",
       "      <td>20.991828</td>\n",
       "      <td>10.070211</td>\n",
       "      <td>29.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>-98.539816</td>\n",
       "      <td>36.628781</td>\n",
       "      <td>413.238281</td>\n",
       "      <td>11.448056</td>\n",
       "      <td>14.993</td>\n",
       "      <td>31.370833</td>\n",
       "      <td>65.004333</td>\n",
       "      <td>2.072126</td>\n",
       "      <td>8.169570</td>\n",
       "      <td>20.766344</td>\n",
       "      <td>9.928065</td>\n",
       "      <td>29.3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    longitude   latitude   elevation        LOD  total_precipitation  \\\n",
       "0 -118.352109  46.929839  524.561829  11.168056                3.362   \n",
       "1 -118.958859  47.150327  451.084656  11.161111                1.536   \n",
       "2 -118.695237  46.811686  427.179199  11.171944                2.472   \n",
       "3 -118.699677  47.162342  521.150330  11.160833                3.528   \n",
       "4 -118.434056  47.157512  571.114075  11.161111                4.317   \n",
       "5 -118.510160  47.006888  499.476074  11.165833                3.671   \n",
       "6  -98.454318  36.508221  394.365906  11.451111               10.565   \n",
       "7  -98.444745  36.650698  401.645874  11.447778               13.368   \n",
       "8  -98.319893  36.702452  359.771393  11.446389               12.767   \n",
       "9  -98.539816  36.628781  413.238281  11.448056               14.993   \n",
       "\n",
       "    minMAT30   maxMAT30  ratioMNDVI30  mean_wind_speed  mean_temperature_diff  \\\n",
       "0  29.588000  62.185833      2.101725         4.202903              17.679355   \n",
       "1  29.681000  63.494667      2.139236         4.044194              17.873925   \n",
       "2  30.269500  63.789667      2.107391         3.991398              18.457634   \n",
       "3  29.487500  62.002333      2.102665         4.630054              17.826935   \n",
       "4  28.834833  60.857667      2.110561         4.717097              17.187527   \n",
       "5  29.496000  61.706000      2.092012         4.679677              17.844516   \n",
       "6  31.529167  65.112000      2.065135         8.180699              20.610538   \n",
       "7  31.417333  65.034000      2.070004         8.170323              20.738495   \n",
       "8  31.339000  65.082167      2.076715         8.287366              20.991828   \n",
       "9  31.370833  65.004333      2.072126         8.169570              20.766344   \n",
       "\n",
       "   std_temperature_diff  yield  \n",
       "0              9.148266   35.6  \n",
       "1              9.295265   35.6  \n",
       "2              9.581113   35.6  \n",
       "3              9.166665   35.6  \n",
       "4              8.905124   35.6  \n",
       "5              9.182010   35.6  \n",
       "6              9.833779   29.3  \n",
       "7              9.912246   29.3  \n",
       "8             10.070211   29.3  \n",
       "9              9.928065   29.3  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_2014_new.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([u'CountyName', u'State', u'Latitude', u'Longitude', u'Date',\n",
       "       u'cloudCover', u'dewPoint', u'humidity', u'precipIntensity',\n",
       "       u'precipProbability', u'precipAccumulation', u'precipTypeIsRain',\n",
       "       u'precipTypeIsSnow', u'pressure', u'temperatureMax', u'temperatureMin',\n",
       "       u'visibility', u'windBearing', u'windSpeed', u'NDVI', u'DayInSeason',\n",
       "       u'Yield', u'Location', u'precipTotal', u'temperatureDiff',\n",
       "       u'temperatureRatio', u'temperatureAverage'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_2013.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([u'CountyName', u'State', u'Latitude', u'Longitude', u'Date',\n",
       "       u'cloudCover', u'dewPoint', u'humidity', u'precipIntensity',\n",
       "       u'precipProbability', u'precipAccumulation', u'precipTypeIsRain',\n",
       "       u'precipTypeIsSnow', u'pressure', u'temperatureMax', u'temperatureMin',\n",
       "       u'visibility', u'windBearing', u'windSpeed', u'NDVI', u'DayInSeason',\n",
       "       u'Yield', u'Location', u'precipTotal', u'temperatureDiff',\n",
       "       u'temperatureRatio', u'temperatureAverage'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_2014.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save features to disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_2013_new.to_pickle(os.path.join('data','df_2013_features.df'))\n",
    "df_2014_new.to_pickle(os.path.join('data','df_2014_features.df'))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
